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ABSTRACT 

Formulas  are  derived  for  the  initial  bias,  variance,  and  spectrum  of  the 
sample  mean  in  finite  state  Markov  processes.  The  focus  is  on  application  of 
such  expressions  to  the  steady-state  simulation  problem. 
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SIGNIFICANCE  AND  EXPLANATION 

"  Various  techniques  have  been  proposed  for  determination  of  confidence 
intervals  associated  with  steady-state  quantities  in  simulation.  Evaluation 
of  such  procedures  requires  comparison  of  their  performance  on  stochastic 
systems  with  known  characteristics.  In  this  paper,  we  therefore  derive 

if  /  j.  n  '  x  ■  .  I  S~r  '3.f  c  Z.  *  f  4-  i *t  -  ■> ■■  "  *  r  '  ; 

computable  formulas  for  several  quantities  associated  with  finite  state  Harkov 
chains,  and  discuss  their  relevance  to  the  steady-state  simulation  problem. 
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SOME  ASYMPTOTIC  FORMULAS  FOR  MARKOV  CHAIMS 
WITH  APPLICATIONS  TO  SIMULATION 

Peter  W.  Glynn 

1.  INTRODUCTION 

Consider  the  simulation  of  a  stochastic  process  {Xfc  :  t  >  0}  for  which 
(1.1)  Xt  — >  X 

(-->  denotes  weak  convergence).  In  many  simulation  applications,  it  is  of  interest  to 
determine  confidence  intervals  for  r(f)  “  Ef(X),  where  f  is  some  real-valued  functional 
defined  on  the  state  space  of  Xfc .  This  problem  is  known,  in  the  simulation  literature,  as 
the  steady-state  simulation  problem,  and  a  great  deal  of  effort  has  been  devoted  toward  its 
solution;  see  Chapter  5  of  Fishman  (1978)  or  Section  8.6  of  Law  and  Kelton  (1982)  for  a 
complete  discussion  of  the  problem. 

The  evaluation  of  simulation  methodology  for  the  steady-state  simulation  problem 
requires  that  one  possess  a  class  of  models  for  which  parameters  of  interest  My  be 
calculated  analytically.  Behavior  of  the  procedures  on  the  models  then  provides  a 
"benchmark"  from  which  to  judge  their  overall  perforMnce.  Our  goal,  in  this  paper,  is  to 
establish  a  variety  of  formulas  for  finite  state  Markov  chains  (in  both  discrete  and 
continuous  time)  and  to  discuss  the  importance  of  these  formulas  in  the  context  of 
methodology  evaluation. 

One  of  the  earliest  techniques  proposed  for  dealing  with  steady-state  simulation 
problems  is  the  technique  of  replication.  The  simulator  chooses  t  large,  and  simulates 
the  process  up  to  time  t,  creating  a  sample  path  (X^  i  0  <  s  <  t}.  The  simulator 
repeats  this  step  a  time*,  creating  a  collection  {X*  i  0  <  s  <  t},  1  <  i  <  m,  of  a 
independent  replicates  of  the  process.  The  parameter  r(f)  is  then  estimated  by 

rt<m,f)  -  —If1  mSd.  • 

*  '  1-1  0  * 
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Hot*  that  th«  independence  of  the  caplleataa  yield*  a  central  Halt  theorea,  and  a 
consequent  asymptotically  valid  confidence  interval.  In  any  case,  the  Man  square  error  of 
the  estiaate  rt(a,f)  is  given  by 

(1.2)  B(rt(a,f)  -  r(f))2  -  o2(rt( 1,f ) )/m  ♦  b2(f) 

where 

bfc(f)  -  8rt(1,f)  -  rtf)  . 

It  is  clear  that  for  large  a,  the  bias  term  bfc(f)  is  the  prlaary  contributor  to  the 
Man  square  error.  As  a  result,  the  initial  bias  term  bfc(f)  has  attracted  a  great  deal 
of  attention  in  the  simulation  literature  t  for  a  survey,  see  Wilson  and  Pri taker  (1978). 
Section  2  is  therefore  devoted  to  formulas  for  bt(f),  and  to  a  qualitative  discussion  of 
initial  bias. 

Nora  recently,  a  variety  of  single  replicate  procedures  have  been  proposed.  They  rely 
on  the  fact  that  for  uny  processes  satisfying  (1.1),  there  exists  a  constant  s(f), 

depending  on  the  process  Xfc,  such  that 

t 

(1.2)  Si  (/  f(x  )ds/t  -  r(f) )/s(f)  — >  N(0,1) 

0  * 

where  W(0,1)  is  a  unit  normal  random  variable  (result  (1.3)  holds,  in  particular,  for 

finite  state  Markov  chains).  Confidence  intervals  based  on  (1.3)  require  consistent 

estlMtors  for  the  constant  s2(f).  In  Section  3,  formulas  are  derived  tor  the  constant 

■2(f),  thereby  allowing  the  study  and  comparison  of  different  estlMtors  for  a2(f). 

These  formulas  extend  the  work  of  Hasan  and  Pritaker  (1980)  on  continuous  tiM  Markov 

chains  with  dlagonalisable  generators  to  the  general  case.  Section  4  is  devoted  to 

solution  of  several  related  conjectures  of  Hasan  and  Pritsker. 

One  well-studied  class  of  estlMtors  for  s2(f)  is  based  on  spectral  techniques.  If 

{Xfc)  is  a  second  order  stationary  process,  it  can  be  shown,  under  certain  regularity 
2 

conditions,  that  s  (f)  ■  2«c(0),  where  e(X)  la  the  spectrum  of  {Xfc>  given  by 

/  eiAtcov(f(X0),f(Xt))dt  . 


(1.4) 


c(l) 


Several  recent  papers  (see  Heidalberger  and  Welch  (1981a),  (1981b),  for  example)  have 
proposed  techniques  band  on  estimating  s2(f)  via  polynomial  fitting  to  an  estimated 
spectrum  in  a  neighborhood  of  sero.  Section  5  therefore  derives  formulas  for  the  spectrum 
corresponding  to  finite  state  Markov  chains. 

Before  concluding  this  section,  it  should  be  noted  that  the  above  discussion  for 
continuous  time  processes  csrries  over,  in  an  obvious  may,  to  discrete  time  processes  - 
this  justifies  the  interest  in  formulas  for  discrete  time  Markov  chains. 


2.  POMWLAB  TOR  THB  INITIAL  BIAS 


Let  {x  <  n  >  0}  be  an  Irreducible  Markov  chain  of  period  d,  with  transition 
ft 

aatrix  P,  on  state  space  S  “  {1,2,..., a).  Such  a  chain  necessarily  has  a  unique 
stationary  distribution  *  «  (*1,«>*,*g|)  solving  wp  -  w.  Given  a  row  vector 
f  “  (f(1),...,f(a))  (f  denotes  the  transpose  of  f),  it  is  well  known  that 


n-1 

i  *  f‘V  * ,f 

k“0 

with  probability  1,  for  any  initial  distribution  y  ”  (M1,...,Ma>  ( ma  *  P{kg  “  i)).  Let 

i  n_1 

“  *  &  I  *<*,.>)  -  «  » 

v  n  k-0  * 

here  B^(  )  stands  for  expectation  under  initial  distribution  ii.  Our  objective  is  to 
obtain  formulas  for  the  initial  bias  b  (u,f). 

ft 

He  will  need  the  following  standard  results  from  Markov  chain  theory  (see  Kearny  and 


Snell  (1960), 
(2.1) 

(2.2) 

(2.3) 

(2.4) 

* 

The  Matrix  r 
that  when  p 


p.  70,  71,  100)i 

p®4  ♦  Ilg  as  n  ♦  where  ng  is  a  stochastic  Matrix 

II  -  (1  ♦  P  ♦  ...  ♦  Pd_1)/d,  where  H  has  all  rows  equal  to  * 

up  -  pn  -  n2  -  n 

e  a  ^  a  a 

the  inverse  Matrix  F  »  (II  -  Q)  exists,  where  Q  -  P  -  X  . 

is  called  the  fundamental  Matrix  of  the  Markov  chain,  it  is  worth  noting 

.  .  •  . 
is  aperiodic,  the  matrix  P  has  the  representation  F  •  £  (P  -  II)*.  Since 


k*0 

the  natural  analog  of  n  for  transient  chains  is  the  aero  Matrix,  it  follows  that  the 
fundamental  matrix  is  a  generalised  form  of  the  potential  Matrix  (see  Clnlar  (1975),  p. 


196-7). 


(2.5)  Theorem.  The  initial  bias  b  (u.f)  is  given  by 

ft 

(2.6)  b  (y,f)  -  y(X  -  Pn)Ff/n  . 

n 

Furthermore,  if  n  ■  kd  ♦  i,  where  0  <  i  <  d,  then 

(2.7)  b  (y,f)  -  y(I  -  P^Hn)Pf/n  «■  0(pn) 

n  u 

where  0  <  p  <  1  (a  sequence  bn  is  0(an)  if  there  exists  K  >  0  such  that 

lb  |  <  K|a|). 
n  n 


-4- 


Proof.  The  bias  can  be  written  in  the  form 


1  n-i  i  n-i  . 

l  l  Uip*  f(j)  -  [  Vjf(j)  *7  E  U(PK  -  n)f 


k-0  i,J 

Now,  it  is  easily  verified,  using  (2.3),  that 


n-1  , 

l  <p  -  nun  -  q)  -  i  -  pn 

k-0 


fro.  which  (2.6)  follows  immediately,  after  postmultlplying  through  (2.8)  by  F.  Equation 
(2.7)  is  a  direct  consequence  of  the  geometric  convergence  of  Pnc*  to  IIq  (see  Corollary 
4.1.5  of  Kemeny  and  Snell  (I960)).  || 

This  result  generalises  Theorem  7-15  of  Heyoan  and  Sobel  (1982)  (their  proof  requires 
that  P  be  aperiodic).  He  now  illustrate  the  application  of  the  theorem  to  a  two  state 


Markov  chain.  Let 


P  -  f1  "  *  *  ) 

1  b  1  -  bJ 


where  a,b  >  0  and  0  <  a  ♦  b  <  2.  Then  {Xr}  Is  aperiodic  and 

*  “  —~—r  (b  a) 


-rrsS  V  ♦  — ■ — j  Ua  V- 

(a  +  b) 


b  (u,f) 
n 


- - — 5-  (an  -  by  )(f(1)  -  f (2))  ♦  0(pn) 

(a  *  b)  n 


A  similar  bias  formula  can  be  obtained  for  continuous  time  Markov  chains.  Let 
(*t  :t  >  0)  be  an  irreducible  Markov  jump  process  on  state  space  E  »  { 1,2, . . . ,m},  with 
generator  Q  (recall  that  Q  generates  Xfc  in  the  sense  that  P(t)  -  exp(Qt),  where 
Pj^(t)  -  P{Xfc  «  j|XQ  “  i})  and  unique  stationary  distribution  w  solving  »Q  ■  0. 
Retaining  the  notatlonal  conventions  previously  stated,  set 

.  t 

b  (y,f)  -  E  f-  /  f(X  )ds)  -  »f  . 


The  following  reaults  are  wall  known 


(2.9)  P(t)  ♦  II,  where  1!  has  all  rows  identical  to  t 

<2.io)  p(t)n  -  np(t)  -  n2  -  n  . 

We  will  also  need  the  following  leaana . 

(2.11)  Lemaa.  The  inverse  matrix  F  •  (II  -  Q)  1  exists. 

Proof.  Using  (2.10)  and  the  fact  that  IIQ  *  0  «  Qll,  observe  that 

t  t 

(2.12)  (II  -  Q)(/  (P(s)  -  n)ds  ♦  n)  -  -  /  Q  exp(Qs)ds  ♦  II  »  1  -  exp(Qt)  ♦  II  . 

0  0 

Then,  letting  lAl  -  ux {  £  |A  |},  we  have 

i  j  13 

•  •  1  • 

/  IP(a>  -  nida  -  l  /  l(P(n)  -  H)P(s)lds  <  l  lP(n)  -  ni 
0  n»0  0  n»0 

which  is  finite,  since  P(1)  is  an  aperiodic  irreducible  matrix  and  therefore  P(n) 
converges  to  II  geometrically  fast.  Renee,  letting  t  ♦  •  in  (2.12),  we  see  that  II  -  Q 
has  an  inverse.  | | 

Initial  bias  for  continuous  tine  Harkov  chains  is  detenained  by  the  following  theorem. 

(2.13)  Theorem.  The  bias  bt(i>,f)  is  given  by 

bt(g,f)  -  |i(I  -  P(t))Pf/t  -  |i(I  -  lI)Ff/t  ♦  DJe’®*)  , 
where  a  is  some  positive  constant. 

The  proof  of  this  theorem  is  similar  to  that  given  in  the  discrete  time  case.  It 
should  be  pointed  out  that  Theorem  2.13  generalises  a  result  of  Crassman  (1982)  given  for 
f (k)  -  k. 

These  initial  bias  formulas  have  several  interesting  properties.  First  of  all,  we 
observe  that  there  exists  a  constant  c(w,f)  such  that 

tk(bt(ii,f>  -  c(u,f)/t)  ♦  0 

for  all  k  >  0.  Hence,  in  any  bias  expansion  of  the  form 

b  (p,f)  -  l  c  (M,f>/t4  ♦  0(t‘r-1)  , 
c  * 


it  nst  be  that  c ^(p,f)  •  0  for  1  >  2.  Secondly,  for  any  it  and  t,  there  exists  t 
such  that  for  a  >  t,  either i 

1)  ba<|i,f)  decreases  aonotonically  to  saro,  or 
ii)  b#(n,f)  increases  aonotonically  to  saro. 

Hence,  for  s  sufficiently  large,  the  Mas  has  a  constant  sign.  Several  proposed  initial 
bias  procedures  require  this  sign  consistency  property  (see,  for  exaaple,  Schruben  (1982), 
p.  577).  Of  course,  the  above  discussion  is  equally  valid  for  discrete  tine  chains 
(provided  one  accounts  for  periodicity). 


VARIAjjCj  FORMULAS  TOR  THX  SAMPLE  MEAN 


A  natural  way  to  try  to  avaluata  a(f)  la  to  take  the  variance  of  both  sides  of 
(1.3),  yielding  the  formal  relation 

(3.1)  t  a2U  f(X  )ds/t)  ♦  s2(f)  . 

0  ‘ 

For  finite  state  Markov  chains,  relation  (3.1)  can  be  justified  rigorously,  in  the  sense 
that  it  is  correct  that 


<3.2)  t  /  «<*8)da  -  wf)2  a2(f)  » 

0 

•  similar  result  holds  in  discrete  time.  For  continuous  time  Markov  chains,  (3.2)  can  be 
proved  by  using  the  fact  that  {Xfc  :  t  >  0}  is  ^-mixing  and  then  applying  Theorem  20.1  of 
Billingsley  (1968).  The  discrete  time  version  of  (3.2)  follows  from  Theorem  3  of  Chung 
(1966),  p.  102.  The  following  theorem  therefore  provides  a  formula  for  evaluation  of 
s2(f). 

(3.3)  Theorem.  Let  {Xn  i  n  >  0}  and  {Xfc  :  t  >  0}  satisfy  the  same  assumptions  as  in 
Section  2.  Then, 


(3.4) 

(3.3) 


«  I  f(Xk)  -  »f)2  -  f'T(I  -  H)f  +  2f'TP(F  -  H)f  ♦  2  f'TJP1**1  -  P)F2f 

k-0  n 

t  *«(*  /  -  »f)2  -  2f *T(F  -  H)f  ♦  7  f’T(P(t)  -  I)P2f  , 

o  "  c 


where  T  is  a  diagonal  matrix  with  T^  ■  w^. 

Froof .  The  process  {Xr}  is  stationary  under  P((>),  so 


n-1 


(3.6) 


4 

■■.4  J 


«V 


»f>  -  vartf(X0> 


,  n-1 

♦  *  I 

n  .  L. 


(n 


kjcov^ftXgbf  (X^)) 


(3.7) 


cov^f  (*0),fUk>> 


-  I  i  ((llP|  f())  -  I  w  *  f(i)f(j>  -  f’T(P*  -  Il>f 

i.j  3  i.j  3 


and 

n- 1  «  n-1  - 

(3.8)  l  (n  -  k)(P  -  Il)<n  -  Q)  -  l  (n  -  k)(PK  -  P  ) 

k"1  k-1 

n  n 

»  nP  -  l  PK  -  n(P  -  m  -  l  (P  -  m  +  I  -  n  . 
k-1  k-0 

Applying  (2.8)  to  the  sun  in  (3.8),  and  postmultiplying  through  (3.8)  by  F  yields 

(3.9)  l  (n  -  k)(Pk  -  H)  -  n(P  -  1I)F  -  (I  -  Pn+1)P2  +  (1  -  H)F  . 
k-1 

Since  II(  II  -  Q)  -  II,  it  follows  that  H  -  HF.  Also,  Q  -  (If  -  I)  (II  -  Q)  so  that 
**  »»2 

QF  -  H  -  I  and  thus  OF  -  (II  -  I)F.  These  observations,  together  with  (3.6),  (3.7),  and 

(3.9) ,  lead  directly  to  (3.4).  The  proof  of  (3.5)  is  sinilar.  1 1 

The  right  hand  side  of  (3.5)  can  be  algebraically  rearranged,  by  using  the  Identity 
F-n-II-dl+Q)1,  to  obtain  Theorem  1  of  Hazen  and  Pri  taker  (1980).  Their 
derivation  required,  however,  that  Q  be  diagonalizable.  The  formula  also  extends 
equation  (16)  of  Grassman  (1982)  to  general  f.  Formula  (3.4)  is  an  exact  form  of  an 
asymptotic  result  found  on  page  84  of  Kemeny  and  Snell  (1960). 

We  now  apply  Theorem  3.3  to  determine  s2(f)  for  the  two  state  Markov  chain  studied 
in  Section  2.  Routine  calculations  show  that 

s2(f)  -  f'T(I  -  ll)f  2f'TP(F  -  il)f  -  "(-2  ~  b)  (f(1)  -  f ( 2 ) ) 2  . 

(a  +  b)3 

We  can,  in  fact,  extend  Theorem  3.3  to  cover  arbitrary  Initial  distributions. 

(3.10)  Theorem.  Let  (Xn  i  n  >  0}  and  {X^  :  t  >  0}  satisfy  the  same  assumptions  as  In 
Section  2.  Then, 
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n  Vi  J  f(xk>  • ,f)  -  ■ 


t  *v{{  /*  -  «f)2  -  s2(f>  ♦  0(i)  . 


Proof.  Wa  prove  only  the  cese  where  {x^}  is  aperiodic)  the  periodic  and  continuous  tis 
proofs  require  only  sinple  Modification.  Let  g(j)  ■  f(j)  ~  »f  and  observe  that 


<3.11)  nlB  (i  l  g(xk))2  -  Ew(i  l  g(xk))2| 


n-2  n- 1 


-  I  *  92<X)-lg2(X)  ♦  i  l  l  (I  g(X  )g(X  >-E  g(X  )g(X  )) 
"  k-0  M  *  *  °  "  k-0  t-k-t-1  W  *  *  *  1 


Now,  Pn  ♦  II  geometrically  fast  (see  Corollary  4#  1.5  of  Ksaeny  and  Snell  (I960))  so  there 

k  k 

exist  constants  o  >  0  and  0  <  p  <  1  such  that  |P  -  s^|  <  op  .  Thus#  for  £  >  k, 

|Eug(Xk)g(Xt)  -  E^glX^glXjH  -  |  J  l^gtjHpJj  -  Vj)  l  (pJ'*  -  *r)g(r)| 

i»  j  r 


<  Igl2  l  |l  « 2p*  -  IglVpV 

i.j.r 


where  Igl  -  max  |g(i)|.  This  inequality,  together  with  Theorem  2.5,  allows  one  to  bound 
1 

(3.11)  by 

,  -  2  ,  n-2  n-1 

|b  (p,g2 ) |  ♦  i  Igl  ■  l  \  a  p*  -  0(1/n)  . 
k-0  l-k+1 

Application  of  (3.4)  completes  the  proof.  I! 

Theorem  3.10  allows  us  to  obtain  an  asymptotic  formula  for  the  mean  square  error  of 
the  estimator  rt(m,f)  used  in  the  method  of  replication.  By  (1.2),  and  Theorem*  2.5, 
3.3,  and  3.10, 


*H(rt(*,f)  -  »f>2  -  I(l(rt{*,9))2  -  j  (IiJ(rt(t,9)>2  -  *2rt(1,9))  ♦  b2(|i,g) 


-  f'TCP  -  1»<  ♦  \  ( lid  -  H)Pf>2  ♦  0(-lr)  i 
t2  nt 


an  analc>90ua  axpanaion  hold*  in  tha  discrete  tine  netting 


SOLUTIOM  TO  OOHJECTORSS  OF  HAZBN  AND  PRITSKBR 


In  their  study  of  continuous  tins  Markov  chains.  Hasan  and  Prltskar  considered  the 

dependence  of  s(f)  on  scaling  of  the  generator  Q.  writing  s(Q,f)  to  indicate  the 

2  2 

dependence  of  s(f)  on  Q,  they  showed  that  if  a  >  0,  then  s  (oQ,f)  -  s  (Q,f)/a  for 
finite  state  processes  and  conjectured  that  the  same  result  holds  for  countable  state 
processes,  as  well.  The  following  theorem  answers  their  conjecture  (see  Feller  (1971),  p. 
326-32  for  definitions  and  results  on  Markov  jump  processes). 


(4.1)  Theorem.  Let  Q  be  an  Irreducible  conservative  (i.e.  >  0  for  i  *  J, 

-•  <  m  ~  1  generator.  If  the  minimal  process  {xfc}  corresponding  to  Q 

satisfies 


t 

P  {(/  (f(X  )  -  r(f ) Ids]  <  x/t  s(Q,f ) }  ♦  p{h(0, 1 )  <  x> 

M  0 

* 

as  t  ♦  «,  then  the  minimal  process  {Xfc}  corresponding  to  aQ,  for  a  >  0,  satisfies 

t  * 

P„K/  <*<*.>  *  r(f))ds)  <  sioQ.f ) }  ♦  p{n( 0, 1 )  <  x} 

M  0 

and  s2(«Q,f)  -  s2(Q,f)/a. 

Proof.  Since  {Xfc }  is  the  minimal  process  corresponding  to  Q,  it  follows  that  it  nay  be 
constructed  via  a  discrete  time  Markov  chain  {Y^}  that  determines  the  sequence  of  states 
visited  by  Xt,  with  the  holding  tine  in  the  kth  state  visited  given  by  an  exponential 
random  variable  with  parameter  q(Yk)(q(i)  -  -Qu>.  On  the  other  hand,  the  minimal  process 

A 

Xfc  associated  with  aQ  has  the  sane  embedded  discrete  time  chain  (Yk>,  but  with 
holding  times  determined  by  exponential  random  variables  with  parameters  uq( Y^).  Hence, 

A  A 

one  can  represent  X  via  X  -  X. ,  so  that 

V  W  OR 


(/  (f(Xs)  -  r(f))ds)/tV2«1/2  -  (/  (f(X  >  -  r(f))ds)/t1/2a1/2 


<*V2(/  (f (X  >  -  r(f))ds]/t1/2  , 

A  • 


from  which  the  theorem  follows.  | | 
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An  application  of  the  result  shows  that  the  variance  constant  s2(Q,f)  for  the  queue- 
length  process  associated  with  an  M/M/1/*  queue  with  arrival  rate  al  and  service  rate 
ay  is  proportional  to  1/a  (see  p.  31  of  Hasen  and  Prltsker  (1980)). 

Before  proceeding  to  the  second  conjecture  of  Hasen  and  Pritsker,  it  is  convenient  to 
discuss  a  second  group  of  formulas  for  s2(f),  based  on  the  regenerative  structure  of 
finite  state  Markov  chains.  The  regenerative  property  dictates  that  blocking  the  sample 
path  of  the  process  according  to  consecutive  entrance  times  Tj  into  some  fixed  state, 
say  i,  yields  a  sequence  of  Independent  and  identically  distributed  random  variables.  It 
is  to  be  expected,  then,  that  the  variance  constant  s2(f )  can  be  evaluated  in  terms  of 
quantities  expressed  over  a  single  regenerative  block.  In  fact,  it  can  be  shown  that  (see 
Smith  (1955),  Theorem  9) 

T1 

(4.2)  S2(f)  -  E  (/  (f(X  )  -  wf)ds)2/E  T 

1  0 

where  (• )  denotes  the  expectation  conditional  on  XQ  »  i  (a  similar  formula  holds  in 
discrete  time;  see  Chung  (1956),  p.  99).  Hordi jk,  Iglehart,  and  Schassberger  (1976)  derive 
matrix-theoretic  expressions  for  the  numerator  and  denominator  of  (4.2).  Prom  a  historical 
viewpoint,  it  la  interesting  to  note  that  there  is  a  third  group  of  formulas  for  s2(f)» 
based  on  the  elgenstructure  of  the  transition  matrices;  sae  Romanovsky  (1970),  p.  241. 

Returning  now  to  the  second  conjecture,  consider  a  capacity  one  single  server  queue 
with  Poisson  arrivals  and  Erlang-p  service  times,  with  inter-arrival  and  service  time  means 
given  by  1/1  and  p/u  respectively.  If  one  is  interested  in  the  variance  constant 

associated  with  the  number  of  customers  in  queue,  then  the  method  of  stages  shows  that  the 

2 

constant  may  be  evaluated  by  considering  *  (f)  for  the  continuous  time  Markov  chain 
described  by  the  (p  ♦  1)  by  (p  +  1)  generator 
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where  f  *  (0,1,1, ...,1)  (l.e.  f  la  ona  so  long  aa  tha  custooer  la  in  aarvica).  Writing 

s'(p,u.f)  to  danota  tha  dapendance  of  s2(f)  on  p  and  u,  tha  conjecture  of  Hasan  and 
Fritsker  was  that  for  it  >  0, 

(4.3)  a2(p,p|t,f)  ■  a*(1,|i,f)  . 

Halation  (4.3)  can  ba  nost  aasily  provad  by  using  (4.2).  bat  Zq.Zj, . . .  ,Zp  be  indapandant 
axponantial  randon  variablaa  with  ZZQ  »  1/X  and  BZ^  *  1/pit  for  i  >  1.  Than, 

(4.4)  *.T,  -  B(Z  ♦  ...  +  Z  )  -  1/X  ♦  1/U 

i  l  0  P 

T1  -  p 

(4.5)  E1(/  (f(*s)  -  «f)daf  -  *i(-»«0  ♦  I  *td  "  »*>) 


<»f)2  -2—  ♦  (1  -  *f)2  -  - - - r 

X2  pit  (X  ♦  u)2 


ainca  *f  ”  X/(X  *  |i).  Substituting  (4.4)  and  (4.5)  into  (4.2),  ona  gats 

a  (p.pn.f)  ”  J  - c  3  * 

p  (X  ♦  p)3 

verifying  (4.3).  Incidentally,  it  is  easily  shown,  using  (4.2),  that  s2(p,pp,f)  tends  to 
tha  variance  constant  associated  with  tha  constant  service  tine  version  of  tha  nodal  as 


p  ♦  •,  as  would  ba  expected 


■N 

% 


jj 

j 


j 

k  ' 

t  -i 

•  • 
* 

I 

t- 
t; 

f 

! 

< 


5.  FOBHPLAS  TOR  TOT  SPECTRAL  DEH8ITY 

The  spectral  density  of  a  discrete  ties  Markov  chain  la  defined  by 

*a>  *2?  £  •Uk«>Vf<V'f(V)  * 

for  continuous  ties  Markov  chains,  c(X)  is  given  by 

c(X)  - -57  /  eiXtcov  (f(x„),f(x  »)dt  . 

the  spectral  density  of  finite  state  Markov  chains  May  be  conputed  via  the  following 
theoren. 

(5.1)  Theory.  Let  (X  }  and  (X  }  satisfy  the  sans  assumptions  as  in  Section  2.  then, 
the  inverse  Matrices  f(l)  -  (I  -  Q  ♦  (e1*  -  1)1)  1  and  P(X)  -  (II  -  Q  +  iXl)-1  exist  for 
all  X,  and  the  spectral  densities  c(X)  and  c(X)  are  given  by 

(5.2)  2ac(X)  -  f  *T(I  -  B)f  ♦  f*TP(P(-X)  ♦  F(X)  -  JI(eiX  ♦  e“lX))f 

(5.3)  2*c(X)  -  f'T(F(-X)  ♦  F( X)  -  211/(1  ♦  X2))f 

Proof.  Me  give  the  proof  in  the  discrete  ties  aperiodic  case,  the  proofs  in  the  other 
cases  being  sladlar.  Using  (3.7),  one  gets 


(5.4) 


2wc(X )  -  f'T(I  -  H)f  ♦  l  {9~iXk  *  elXk)fT(Fk  -  n)f  . 
k-1 


How,  observe  that 


(5.5)  l  e1Xk(Pk  -  n)(n  -  P  ♦  e~1XI)  *  P  -  H  -  #iXn(pn+1  _  j|)  . 

k-1 


Also,  it  is  evident  that 

l  leiXk(Pk  -  11)1  <  l  lpk  -  ni  <  • 
k-1  k-1 

since  Pk  ♦  H  geoawtrically  fast,  and  thus  the  sum  in  (5.5)  converges  to  none  Unit,  say 
D(X).  Taking  Units  in  (5.5)  yields 

D(X)(H  -  P  ♦  a"iXI)  -  P  -  II 
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•o 

and  thus  F(-X)  »  (n 
F(-X)  and  letting 


a 


iX 


-  e  ♦  (« 


-iX 


(D(X)  +  ixn  -  p  ♦  e_1*l)  -  1 
-  1)1)  1  exists.  Postmultiplying  through  (5.5)  by 


n  ♦  •  proves  that 


(5.6) 


l  elXk(pk  -  m  -  (p  -  n)F(-x)  . 

k-1 


*  iX 

It  is  easily  verified  that  1IF(X)  ■  e  H  and  combination  of  (5.4)  and  (5.6)  leads  easily 
to  (5.2).  || 

Formulas  (5.2)  and  (5.3).  together  with  Theorem  3.3,  prove  that  2*c(0)  «  s2(f)  (see 
(1.4)),  justifying  the  use  of  spectral  methods  for  finite  state  Markov  chains.  Returning 
to  the  two  state  Markov  chain  introduced  earlier,  the  computation  of  c(X)  is 
straightforward,  given  that 


F(X)  - 


•lX(b  ♦  a  -  1)  ♦  e12X 


(b  *) 

'b  a' 


(e1X(b  ♦  a  -  1)  ♦  e12X)(a  +  b) 


^e1X(a  ♦  b)  -  b 
-b 


eiX(a  +  b)  -  a 


The  formulas  also  yield  same  interesting  qualitative  information  about  the  spectrins  of 

* 

finite  state  Markov  chains.  Applying  Cramer's  rule  to  compute  the  Inverse  matrix  F(X) 

m 

shows  that  the  elements  of  F(X)  are  always  rational  polynomials  in  the  indeterminate 
eXX  -  in  fact,  the  polynomials  describing  the  nuamrator  and  denominator  must  be  of  degree 
less  than  or  equal  to  a.  Consequently,  the  spectrum  of  a  stationary  discrete  time  finite 
state  Markov  chain  corresponds  to  that  of  a  finite  order  autoregressive  moving  average 
process. 
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